Background Estimation from Photographs with Application to Ghost Removal in High Dynamic Range Image Reconstruction

We address the problem of reconstructing a scene background from a set of photographs featuring several occluding objects. We assume that the photographs are obtained from the same view point and under similar illumination conditions. Our approach consists of defining the scene background as a labeling over the set of input exposures, then assigning each labeling a cost, and finally minimizing the associated cost function. The cost function penalizes deviations from the following two model assumptions: background objects are stationary, and background objects are more likely to appear across the input exposures. We approximate object stationariness using a motion boundary consistency term, and object likelihood using kernel density estimations on a decorrelated color space. Both estimates are combined using an entropy-based weighting function. Furthermore, we constraint our solution space in order to prevent partial classification of occluding objects as background, thus avoiding cutting through objects. The cost function is minimized by using graph cuts, and the final result is composed using gradient domain fusion. Our method benefits from the appropriate selection of color spaces for each task. We apply our method to the recovering of clean, unoccluded shots of crowded public places. We also demonstrate the application of our method to the removal of ghosting artifacts during the reconstruction of high dynamic range images from multi-exposure sequences. Our contribution is the definition of an automatic method for consistent background estimation from multiple exposures with occluders, and its application to ghost removal in high dynamic range image reconstruction.